Color-Aware Regularization for Gradient Domain Image Manipulation

We propose a color-aware regularization for use with gradient domain image manipulation to avoid color shift artifacts. Our work is motivated by the observation that colors of objects in natural images typically follow distinct distributions in the color space. Conventional regularization methods ignore these distributions which can lead to undesirable colors appearing in the final output. Our approach uses an anisotropic Mahalanobis distance to control output colors to better fit original distributions. Our color-aware regularization is simple, easy to implement, and does not introduce significant computational overhead. To demonstrate the effectiveness of our method, we show the results with and without our color-aware regularization on three gradient domain tasks: gradient transfer, gradient boosting, and saliency sharpening. 1 Motivation and Related Work Gradient domain manipulation is the cornerstone of many image processing algorithms from image editing to texture transfer to image fusion. For an overview of gradient domain algorithms and applications we refer readers to [1]. As the name implies, gradient domain algorithms do not operate in the 0th order domain (i.e. color domain), but instead impose changes to the 1st order derivatives of the input image, i.e. the image gradient. When left unchecked, gradient domain processing can result in noticeable color shifts in the 0th domain output image. To ameliorate color-shifting artifacts, most gradient domain approaches impose an additional 0th order constraint either at the boundary of the processed region or over the entire region. Early gradient domain processing approaches (e.g. [2–5]) were formulated using the Poisson equation (see [6]) which incorporates a 0th order boundary constraint on the solution, i.e. the Dirichlet boundary condition. While generally sufficient for most processes, this method can, from time to time, exhibit very noticeable color shifts inside the processed region. As a result, other approaches, especially more recent ones (e.g. [1, 7–11]) impose a regularization over the entire 0th order solution. This is typically done using an L2 norm regularization on one or more of the 0th order image channels. This solution results in a biobjective function that tries to manipulate the image gradient while minimizing 2 Fanbo Deng, Seon Joo Kim, Yu-Wing Tai, Michael S. Brown